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The net result of the social-scietttific developments 
in the 19th and 20th centuries is that educational researcli has 
?.nherited a science that is assumed to constitute a disinterested 
•■echnoloay of social engineering and a benevolent source of positive 
social advance. Onf ortunately, the social ejtperiiikent conducted on 
♦■raditional lines depends upon the extent to which control can be 
applied to the investigation: yet the achievement of such social 
control, through the manipulation of "haves** and "have nots" may come 
into conflict with new assumptions concernim social justice and 
democratic freedom. The problem of truth or„ to put it another way, 
the problem of uncertainty, i s also ever-present in social research. 
The debate over these issues will continue, but two ideas should be 
kep*- in mind; f1^ an evaluation is an intervention in a debate whose 
*erms of reference are never quite the same as those addressed by the 
evaluation: and (2\ the context of an evalmtion or case study is not 
the same as the context of the action that mla.it flow from it. 
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My role this inomirg is, I understand, to provide a prologue 
to the ensuing, more specific papers* VThat I have to say, then, 
relates to a number of topics* These include the purposes of 
educational research; the role of social science in modern society; 
and the significance of mothodological debates in science. Overall, 
my aim is to sketch in some of the context that gives meaning to 
today's deliberations,^ 

My reasons for adopting this v/ide-ranging stance are two-fold. 
First, it helps me to underline the fact thay many of the issues raised 
about educational evaluation are also important to educational inquiry 
in general; and secondly, it enables me to txeat evaluation questions 
not merely as matters of methodology but also as matters of ethics and 
epistemolcgy. 



Paper presented at the Second Annual Minnesota Evaluation Conference 
on 'case study methodology in educational evaluation' , Minneapolis^ 
15th May, 1981. To be published in the conference proceedings 
(in preparation} . 





Innovation and Change 



As a visitor to this country I never cease to be amazed by what 
appears to be a constant drive towards innovation in all spheres of 
life. Innovation is celebrated as a new fronticjr that^ itself^ is 
in a state of permanent renewal. Where I come from a different 
story obtains. Innovation is deemed a maverick^ up-start activity, 
British innovators occupy the shady margins rather than the leading 
edges of society. Indeed^ 'entrepreneur' is such an unspeakzJ^le 
title that the British had to turn to the French to find a label for 
it. 

My contrast between conservative Britain and free-booting America 
is certainly overdrawn^ though I think there is a real sense in which 
the innovation market and the turn-over of ideas is taken for granted 
in the USA, But there is another side to such an innovation- led 
economy. Does innovation necessarily promote change? Or is it 
the .case that^ under conditions of rapid innovation^ change is more 
an illusory phenomenon - a social sleight of hand created through 
the sacrifice of substance to appearance ^ and fabric to fashion? 
Further^ if this illusion is successful^ are consumers lulled into 
a state of false consciousness? Do they assiune themselves to be 
the beneficiaries of innovation? Or are they^ in fact^ its 
xinwitting victims? 

Let me bring my argument a little closer to education. Do the 



changes we talk about really reflect some kind of directional shift 
. towards a deeper or more immediate appreciation of educational 
phenomena? Or are they merely novelties that have been produced 
for sale on the academic market place? is the topic of this 
conference y and oiar presence here today ^ the outcome of relatively 
random forces?, are we just promiscuous follov;ers of fashion^ 
pursuing case study today^ structural equations tomorrow^ and 
something else the next day? 

For my part I am sympathetic to such cm analysini- but I do 
not believe that it provides the whole story. Day-to-day shifts 
in education may appear to bo prompted by random or 'local' 
causes but, in the long run^ a number of rviore profound if not 

.irreversible factors also plays a part. That is^ the dynamism 

/ 

of a nation's schooling is alsoy affected by changes in a nation's 
economic structures^ political/systems^ religious alfiliations^ 
cultural identities y and so on. 
Doing science or doing justice? 

/ 

/ ^ 

Today I'm going to focus upon the relationship between educational 
inquiry and some of these broader shifts. The voices that I am 
goint to take seriously are those which ^ over the last twenty years 
or more^ have argued the case for greater justice in social action. 
The burden of my argument is that the search for 'alternative 
approaches' in evaluation is part and parcel of the same movement. 



Above nil, 1 think the moot impojrtant contribution made by recent 
, evaluation theory to the mainstream of educational inquixy has been 
an increased awareness that • doing science' is one thing, while 
'doing justice' may be something elae. 

The european scientific revolution of the seventeenth century, 
with which we associate people like Francis Bacon and Isaac Newton, 
was premised on the assumption that nature could be Induced to give 
up her truths if tackled in a new wayi that is, through observation 
and experiment. The scientific oJ: empirical method as we know it 
today can be traced back to that era. It constituted a new technology 
that claimed to be more powerful and efficient than cinything previously 
envisaged. Mankind was no longer regarded as being in harniony with 
cnature. Henceforth, the relationship was to be one of domination and 
subordination. 

The extension of mcinkind's dominion over nature was the promissory 
note issued by Bacon and his contemporaries. They believed that the 
scientific revolution would usher in a period of great social wealth 
and even greater personal and political freedom. In its most 
democratically-organised form the scientific revolution promised a 
heaven on earth for everyone. The social budget^p by its protagonists 
assumed that technical efficiency would lead to the creation of a material 
surplus which, in turn, could be xTedistributed to alleviate misery and 



poverty for time. 



The SoctolrSclGntlflc Paradox 

The aspiration to control nature, howiver, also triggered a 
related sentiment - that a comparable technology could be developed 
to control the workings of civil society. The nurturance of this 
idea was a key feature of the 18th century Enlightenment; and its 
maturity in the 19th century was intimately tied up with 
the emergence and establishment of such disciplines as economics, 
sociology, psychology and, not leasts education. 

With the wisdom afforded by hind-sight, I think we now recognise 
that the social-scientific irovolution vms baflt upon a paradoxical 
core assumption - that tlie purpose of the social sciences was to 
extend manlkind* domination over mankind. Before the nineteenth 
century such an assumption was unproblematic: a large propor'-.-^.on 
oi society was denied the civil rights enjoyed by the remainder. 
Thus, the original purpose of the social sciences was perfectly 
consonant with a 'two nations' image of society. Accordingly, 
social scientists were able to develop a rationale and technology 
which f explicitly or implicitly, took only indirect accovmt o:. the 
feelings and aspirations of those whose lives they intended to reform^ 

It was a time when the expert 'knew best* and a time when ^V;e 
administrative solutions of the experts were deemed to be »in the 



beat Interests' of their cllonta. The juatice of nuch methods 

was underwritten by an appeal to the principle that what is good 

for society is aloo good for every individual; and the plausibility 

of such methods was assured by the fact that they operated within 

expanding national economies that were always able to provide a 

3 

little extra for everyone. 

Scientific Method as Problematic 

The net result of these nineteenth and twentieth century 
movements is that educational research has inherited a science that is 
assumed to constitute a disinterested technology of social engineering 
and a benevolent source of positive social advance, 

1 am sure that it will be no great surprise to you that I am 
sceptical of this ahistorical view of science. In particular, I 
discount its claims to have found a set -of universal or absolute 
standards - "the" scientific method - against which other forms of 
inquiry are judged and, usually, found wanting. My own view 
is that any methodology like any course of action, necessarily 
has an ethical dimension and that, accordingly, if we chcinge our 
ethical criteria we are inexorably led to a reconsideration of our 
practice. 

Thus doing evaluation is not the application of a universal 
method but, rather, a constraint striving to resolve - for every new 
occasion - the essential tension that suffuses our concept of social 
science. And it is this striving that is one of the subtexts of 
today's conference. 



Ut me amplify my argument by taking a oloaer look at the 
•universal or abaoluto atandarda' that I reforrod to a moment 
ago. In particular, l would like to demonatrato certain problematic 
features of our current uao of the concepta of efficiency, truth 
and objectivity in research. 

Efficiency and tho 'True' Experiment 

Nearly twenty years ago Donald Campbell and Julian Stanley 

focused on the problem of efficiency in their seminal paper on 

'Experimental and quasi-experimental designs for research on 
4 

teaching' . Their work, quite explicitly, was an attempt to 

extend the rationale of biological experimentation into areas where 

experimental control is less than complete. Two features ofCampbell 

& Stanley's paper are pertinent to my argument. First, it was a 

fomal analysis, having more regard to the design of experiments than 

to their implementation; and secondly, through its discussion of 

5 

external threats to validity , it noted thatthe results of 'true' 
experiments may be open to multiple interpretation. Put another 
way, their paper was about the reduction of uncertainty in the 
social sciences, not its elimination, 

A few years later Campbell revisited these earlier ideas in 

a paper on 'Reforms as escperiments ' . ^ His particular interest was 

the contribution that the social sciences might make to the 'Great 

Society' legislation of the 1960s. His paper concerned itself with 
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the 'political -realities' as well as the technical logic of 
experimentation. And, by the same token, it was addressed directly 
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to tha refojrinlncj ndmlnlatiratpra (or 'aooial englnooiraP ) 
of tJiot epoch. 



In tJ\o Gourao of hie. argumont Campboll ran up arjalnat a gonoral 
problem in oxporimontnl ronoarch i tho ooclal oxporimont conductod 
on traditional linea daponda upon the oxtont to which control or 
rondomlaation can be applied to the context of tho InvQatigation; 
yet, tho achievement of auch aocial control, through the manipulation 
of 'havoa' (the treatment group) and 'have nota' (tha control group) 
may come into conflict with new (or Great Society) assumptiona 
concerning social justice and democratic freedom. 

In the late 1960s, then, Campbell noted the intractable nature 

of the problem but, at the aarae time, came to the defence of the 

experimental model when he claimed to have identified a set of 

experimental conditions wherein randomisation was not a variance 

with social justice. That is, he had found a situation where, even 

in Great Society terms, the efficient experiment is the same thing 

as the just experiment. The particular design requirements noted 

by Campbell relate to the allocation of "social ameliorations' that 
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are in "short supply". Campbell argued that under such conditions 
it would be impossible for everyone to receive the benefits of the 
treatment and that, therefore, randomisation was both technically 
and ethically defensible , 



Campbell's a. .nent - based on a discussion of the Salk vaccine trials- 
is very plausible, but it is flawed on at least one count: the Salk 
trials are not a general instance. On that occasion, the experimental 
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aubjaota reoaivad a tro«t:nient of proaumoti benefit, whereas th(3 
control HubjooUB wore deprlvad of aoniothing - immunity to polio - 
which in fact, thoy l\acl iwyor enjoyod. Thus, contrary to hia 
claim, Campboll'n oxampl<\r only ombracoa infitnncofi whoro bonofits- 
are randomly wlth-hold, not inatancon whoro bonoCita nro randomly 
withdrawn. In tho second cnno, of courao, the doprivatlon in much 
inoro viniblo to the deprived. 

For these reasons, then, I would expect randomisation experiments 
involving incroanes in grant aid to provoke a different consumer 
response from tliose than entail cuts in grant aid. In short, 
the research designs proposed by Campbell may be logically 
equivalent, but they are not socially equivalent. Despite his 
claim that 'this is the ideology that makes possible "true 
experiments" in a large class of social reforms' /° Campbell's 
attempt to reduce social justice to social efficiency must 
be regarded as unsuccessful. 



The net result of Can^ibell's efforts, of course, is that a 

twentieth century tension «s ^ resolved in favour of a nineteenth 

century conception of the relationship between science and justice. 

That is, 'organisational efficiency' and the 'scientific organisation 
11 

of society'- are gtill deemed to be at peace with the interests of 
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the individual. 



In lAfcer work Campbo.ll h«a drawn hack from soma of ]\ia 

oarller poaUlonH and Ima boan loaa ready to Hottle in favour 

of managorial modola of control and off lolajjcy. For inaUanco, 

in a paper dlroatly related to the thomea of thlu conference - 

"•Degreea of freedom" and the case atudy'^^ - ho has dlacuaHod 

the vaiuo of adopting procedures which, unlike tixoaa of 'true' 

exporlmentatlon, deliberately allow 'innumerable alternative 
13 

solutions' . 

But Campbell's new-found support for case study methods still 
relied unilaterally on technical criteria. His argument was that the 
convergent nature of experiments means that only a relatively limited 
number of hypotheses can be examined whereas, by its multi-dimensionality, 
a case study allows many more explanatory hypotheses to be 
sctutinised, including those that were not envisaged at the outset 
of the research. 

Truth in Research 

But, you might ask doesn't the possibility of alternative solutions 
also include the possibility that no one solution will emerge as 
'the' explanation. And doesn't this possibility in its turn, undermine 
the expert status of the researcher? This question brings me to the second 
issue I raised earlier - that of truth in research. 
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Th« problem of tvuUh ov, to put. it .nothor w^y, thu prahlar« 

CKl«(:a, «fl i «i,aw0d u.rUov, in , i-:h« o.r.j«ni«kt.lon«n.1 in,.,. p,-a. .it i..a 
otf "t:ruo" «Kporim«nt-n. in «n i.npnnnnfc «onMo, how^vnr, 11,.. 
i.ncort«in(.v c1o«« not-. ro«hi., in th. pluM.n.non. , |.vt, w« nr., t . yi ,,., 
to atudy but:, in tho Wity wu v«luo thorn. 

hot mo oUor an «xa,n,.l i,„n.Mno two ym>nr; olviXdv.m, .,,,0 
tlmt cnn count up to fivo and tho othor that can count u„ ,0 fn,. 
SiK wooks later tl)o Irirnt one haa movod up to novc., n m .nd 
the older one has moved up to thirteen. The queutlt,,, . uat might, 
bo aakod of theao children is -which one hau m«do tl.o n,o.t progro.u?.. 
At one level this queaUon ia absurdly simple-, but nt another 
the answer given can only be arbitrary, as this example suggests, 
the facts do not speak for themselves, they merely beg furUier 
questions. ,In the instance cited, any answer will hinge upon 
whether the initial differences between the children should be 
taken into account, i^at is, it depends on the moral valuations 
that we bring to the task. Do v. , creat the children and their 
circumstances as different or equal? How, in technical terms should 
we 'weight' them?^^ 



The traditional reactdon of educational researchers to the challenge 
of these ard other empirical uncertainties has been to treat then 
as problems of technical rather than ethical clarification. it 
is assumed that they can be solved by using a higher level of 
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methodological sophistication (e.g. by using multivariate rather 
than bivariate statistics). The net result, of course, has been 
the pursuit of a spurious precision in research; itself driven 
by the popular but niistaken belief that the problems of research 
can be reduced to the realm of methodology and technique. 

Such activities are easily driven forward. Their technical 
precision susstains an illusion of certainty. But, as in life, 
there is always the risk that procedural consensus will break 
down, pulled apart by tlie different valuations adopted by the 
protagonists. At first glance, this pc^sibility seems remote. Yet 
it is the very issue that has fueled the debates and recriminations 
surrounding the Head Start and Follow T^hough evaluations. 

Whenever we hear the call fox results do we have sufficient 
resources to back up Bacon's promissory note? Are our procedures 
as 'hard' as we would like to imagine? Does our delivery record 
match up to our funding talk? I think not. m practice we operate 
with standards that are more double than absolute. Our professional 
misgivings about the certainty of our procedures are, moae often than 
not, expressed as public silences. Our audiences, therefore, still 
expect us to furnish a kind of 'out there', theory-free truth which, 
increasingly, we know we cannot provide. Must we continue in our 
cover-up? Or can we offer an alternative view of the social sciences? 



Objectivity and the Independent Observer 



The professional misgivings and public silences that relate to 
conventional notions of truth also apply to objectivity - the last 
issue that I would like to address tills morning. If education, like 
physics, is regarded as an exact science e»en objectivity and exactltiude 
are taken to be synonymous. Researchers are assumed to occupy a 
privileged vantage-point - one that enables them to render accounts 
of the world that are uniquely free of error or bias. Hence, in 
these terms, a drift towards subjectivity is, at the same time, a drift 
away from the truth. But physics provides us with an alternative 
perspective on truth and objectivity, since the publication of 
Einstein's Special Theory of Relativity, in 1905 it has been more 
readily appreciated that any description of the world is dependent 
upon the time/space location of the observer. Thus, the objectivity 
of an observation, like its truth, is always dependent upon the personal 
knowledge or, what amounts to the same thing, the ethical standpoint 
of the observer. 

p. 

These assertions of a physicist about the convergence of truth 

and subjectivity may seem a little remote from the hurly-burly 

of education inquiry. But, again, they served as an important (if 

unci ted) precurser of Donald Campbell's paper on 'Qualitative knowing in 
16 

action research' that was read to the American' Rsychological 
Association In 1974, The purpose of Campbell's address was to revisit 



some of the issues that had been left out of his earlier discussions 

of 'quantitative-ejcreriraental' approaches to social research. What 

is even more important to this conference is that he included 
17 

•case study' as contributing towards the 'qualitative knowing' 
of the social sciences. 

Set against Campbell's earlier papers on 'Experimental and 

quasi-experimental design' and 'Reforms as experiments', 

'Qualitative knowing in action research" has two noteworthy fefatures. 

First, it disavows managerial models of control by counterposing 

the value of "participant evaluation"; and secondly, it takes the 

convergence of subjectivity and truth to such a point as to 

suggest that the experiences of 'participants" are more "valid* 

18 

than those of • outside observers' , Overall, Campbell's 
•qualitative knowinc/' paper is significant in that it demonstrated^ 
for him at least, a hitherto unacknowledged av/areness of the 
essential tension that pervades the social sciences. Whereas Campbell^ 
initial methodological strictures arose from an apparent concern 
with efficiency, by 1974 the concern for social justice was more 
obtrusive. But, to this day, the problem of combining a » top-down' 
model of efficiency with a 'bottora-up» model of justice, seems 
to have proved as intractable to Campbell as the earlier problem of 
divining the 'true' ejcperiment* 

Towards an Alternative Social Science 



Let me pause for breath. So far in this paper I have identified 



a tension, that, I believe, is inherent to social inquiry 
in general and educational inquiry in particular, I've noted 
too that working in the context of expanding economic systems 
social scientists have been successful in drav/ing J technical 
veil over these issues. 

Today, I have tried to pull back a little of the veil, I 
have questioned some of the assun.ptions that have guided social 
science practice for many years. Together they underwrote a 
science - a technical-ethical calculus - that, somehow, was deemed 
to be beyond society. Hence when the science railed, it was 
the earth-bound practitioners who were blamed, "You didn't do 
it right' is the charge that was made against the* But it is not 
our world that fails to live up to our science- rather. It is 
our science that fails to live up to our world. 

Where, then, does that leave us? I suspect that if you came 
to this conference worried about methodology you are feeling even 
less comfortable by now. You came looking for solutions and all 
I have been able to offer are irresolvable ambiguities and infinite 
regressions. First you saw black boxes in research , now they 
have turned into black holes. But, take comfort, you are not being 
deluded. 



a very real sense, methodological debates are no such thing. 



That Is^ they are not purely about technical matters. Instead^ 
they are driven by all kinds of theoretical, fethical and social 
concerns. Indeed, the lack of consensus that create J such a 
debate is, I believe, a reasonable response to tlie many different 
ways in which the efficiency/justice tension can be resolved in 
the social sciences. But where do these ideas spill over into 
the realm of "case study methodology in educational evaluation'? 
Let me to conclude, suggest two points of departure. 

First ^ I find it useful to consider an evaluation - formal 
or informal - as an intervention in a debate whose terms of 
reference and criteria of worth are never quite the same as those 
addressed by the evaluation. This may arise because the. evaluation 

o 

only addresses a subset of issues or because tlie debate, itself, 
is constantly changing. Either way, the evaluation can never replace 
the debate; that is^ it can only be a sensitizing device. It 
always remains 'an» evaluation, not 'the' evaluation. 

And second, I find it useful to remember that the context of 

an evaluation or case study is not the same as the context of the 

action that might flow from it. That is, other circumstances and 

other audiences may intervene. Necessarily, then, a case study 

evaluation can only prefiture 'a» course of action, not 'the' course 
19 

of action. If we try to offer any more than these gentle promises 
we too, may be guilty of a doiable standard in our work. 



In preparing this paper, I worried whether I was devoting enough 



space to case study and evaluation. Now I suspect I've aeid 
too much. Nonetheless, I hope my remarks provide a s.iiteible back- 
cloth to the remaining papers. My belief is that there are 
substantial links between the general topics of my paper and 
the current concerns of evaluation research and case study 
inquiry. I hopor therefore, that what I have said contributes 
towards building a bridge between your ovm expc^'iences and those 
that are to be examined in the presentations that follov/. 
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